But, also, it has become focus of study because it is one of few theoretical settings that exhibit overparameterized learning .

The Order Oracle has the capability to compare two functions; however, in contrast to the zero-order oracle, it lacks the ability to calculate or utilize the actual value of the objective function. This concept closely mirrors the challenges encountered in real-world black-box optimization problems.

Frequently, the burgeoning field of black-box optimization encounters challenges due to a limited understanding of the mechanisms of the objective function. To address such problems, in this work we focus on the deterministic concept of Order Oracle, which only utilizes order access between function values (possibly with some bounded noise), but without assuming access to their values. As theoretical results, we propose a new approach to create non-accelerated optimization algorithms (obtained by integrating Order Oracle into existing optimization "tools") in non-convex, convex, and strongly convex settings that are as good as both SOTA coordinate algorithms with first-order oracle and SOTA algorithms with Order Oracle up to logarithm factor. Moreover, using the proposed approach, . And also, using an already different approach we provide the asymptotic convergence of . Finally, our theoretical results demonstrate effectiveness of proposed algorithms through numerical experiments.

But, also, it has become focus of study because it is one of few theoretical settings that exhibit overparameterized learning .

This problem configuration encompasses a broad range of applications in ML scenarios, e.g.

In this section we provide a proof of Theorem 9 and a numerical comparison with the P AC-Bayes-Bernstein inequality. The proof is based on the standard change of measure argument. The second ingredient is Bennett's lemma, which is a bound on the moment generating function used Now we are ready to prove the theorem. Therefore, for µ < 0 .5 we have null In this section we provide technical details on minimization of the bounds in Theorems 12 and 15. As most of the other P AC-Bayesian works, we take π to be a union distribution over the hypotheses 14 in both cases.

The bound is based on a novel parametric form of the Chebyshev-Cantelli inequality (a.k.a.

We consider a line-search method for continuous optimization under a stochastic setting where the function values and gradients are available only through inexact probabilistic zeroth and first-order oracles. These oracles capture multiple standard settings including expected loss minimization and zeroth-order optimization. Moreover, our framework is very general and allows the function and gradient estimates to be biased. The proposed algorithm is simple to describe, easy to implement, and uses these oracles in a similar way as the standard deterministic line search uses exact function and gradient values. Under fairly general conditions on the oracles, we derive a high probability tail bound on the iteration complexity of the algorithm when applied to non-convex smooth functions. These results are stronger than those for other existing stochastic line search methods and apply in more general settings.

Frequently, the burgeoning field of black-box optimization encounters challenges due to a limited understanding of the mechanisms of the objective function. To address such problems, in this work we focus on the deterministic concept of Order Oracle, which only utilizes order access between function values (possibly with some bounded noise), but without assuming access to their values. As theoretical results, we propose a new approach to create non-accelerated optimization algorithms (obtained by integrating Order Oracle into existing optimization "tools") in non-convex, convex, and strongly convex settings that are as good as both SOTA coordinate algorithms with first-order oracle and SOTA algorithms with Order Oracle up to logarithm factor. Moreover, using the proposed approach, we provide the first accelerated optimization algorithm using the Order Oracle. And also, using an already different approach we provide the asymptotic convergence of the first algorithm with the stochastic Order Oracle concept.